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1.  Technical  Problem 

We  wish  to  extend  our  work  on  VLSI  implementation  of  learning  neural  networks  to  the 
prototype  system  level. 

2.  Methodology 

To  build  experimental  prototype  learning  systems  we  will:  A)  Develop  a  prototype  of  an 
enhanced  neuron/synapse  chip  using  some  ideas  that  we  have  gained  from  our  existing  chips.  B) 
Develop  a  prototype  VME  based  experimental  platform  for  the  above  devices.  C)  Write 
experimental  prototype  system  software  to  run  the  above  prototype  boards  and  chips  as  co¬ 
processors  for  a  typical  computer  system  such  as  a  SUN  4.  D)  Develop  new  algorithms  to 
perform  other  types  of  learning  suitable  for  prototype  VLSI  implementation. 

3.  Technical  Results 

3.1  System  Level  Hardware 

Redesigned  prototype  learning  chips  were  fabricated.  The  new  prototype  VME  boards 
containing  these  chips  have  been  integrated  with  our  prototype  learning  system.  We  have  also 
integrated  off-the-shelf  analog  input  boards  into  the  learning  system.  A  single  prototype  VME 
board  containing  the  controller  and  a  multichip  neural  system  has  also  being  designed.  For 
further  details,  see  section  1  of  the  quarterly  report. 

3.2  System  Level  Software 

Software  modules  to  interface  with  our  prototype  system  have  been  written.  The  module  in  the 
Braincore  that  interfaces  with  the  prototype  VME  boards  has  been  modified  to  accept  analog 
inputs.  For  further  details,  see  section  2  of  the  quarterly  report. 

3.3  Algorithms 

Theoretical  and  simulation  experiments  were  carried  out  to  gauge  the  efficiency  of  one-weight- 
at-a-time  vs.  parallel  perturbations.  For  further  details,  see  section  3  of  the  quarterly  report. 
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1.  System  Level  Experimental  Prototype  Hardware 
1.1  Accomplishments 

We  have  built  several  experimental  prototype  VME  boards  to  enable  use  of  our  previously 
designed  prototype  learning  chip[1]  as  well  as  some  newly  designed  prototype  learning  chips  as  a 
coprocessor  for  a  SUN  4  computer.  These  prototype  boards  include  a  digital  pattern 
presentation  board,  a  neural  network  board  containing  a  single  learning  chip,  and  a  VME  control 
board. 

We  have  designed  and  fabricated  an  experimental  prototype  synapse-only  chip  which  is  used  for 
interconnecting  the  previously  designed  prototype  neuron-containing  chips.  This  chip  was  fully 
tested  and  works  to  our  requirements. 

Because  of  the  success  of  the  prototype  synapse-only  chip  design,  we  incorporated  its  synapse 
design  into  a  redesigned  prototype  neuron-containing  chip  to  facilitate  system  experiments.  This 
device  was  redesigned  with  some  insights  gained  from  our  previous  learning  chip  and  contains 
faster  and  smaller  synapses.  The  circuits  have  been  modified  and  we  were  able  to  obtain  higher 
yields  than  that  were  obtained  in  earlier  runs.  This  new  prototype  chip  was  fabricated  on  a  finer 
line  CMOS  technology  but  has  been  tested  and  found  to  work  well. 

These  experimental  prototype  learning  chips  were  incorporated  into  the  aforementioned 
experimental  prototype  VME  boards  and  integrated  with  the  prototype  learning  system.  The 
system  is  now  functioning  as  a  co-processor  for  a  SUN  4  computer.  One  aspect  of  this  system  is 
that  it  consists  of  three  different  prototype  VME  boards.  We  have  designed  a  prototype  two 
board  system  where  the  controller  and  a  multi-chip  neural  network  resides  on  a  single  card.  This 
will  make  it  easier  for  us  to  do  multi-chip  learning  experiments  since  the  inter-board  cabling  is 
eliminated. 

We  have  incorporated  commercially  available  D/A  boards  into  our  prototype  learning  system. 
This  has  given  us  the  capability  to  experiment  with  analog  signals  in  various  learning 
applications,  adaptive  equalization  in  particular. 


We  have  carried  out  some  experiments  on  the  prototype  co-processor  system.  We  have  tried  the 
non-linear  adaptive  equalization  problem,  which  is  useful  for  wireless  and  portable 
communication  devices.  The  neural  network  is  trained  with  noisy  data  through  a  dispersive 
linear  channel  and  is  required  to  reconstruct  the  original  data.  Results  are  promising  and  this 
work  has  been  published.^ 

2.  System  Level  Software 

2.1  Accomplishments 

We  have  written  a  module  for  our  Braincore  simulator  to  interface  with  the  above-described 
prototype  VME  boards  to  demonstrate  fast  learning  for  our  prototype  single-chip  system.  This 
module  has  been  modified  to  process  analog  signals  through  an  off-the-shelf  VME  D/A  board. 
This  allows  us  to  demonstrate  and  perform  experiments  using  analog  signals  feeding  directly 
into  the  neural  network  chips. 

The  entire  Braincore  code  was  updated  to  be  ANSI  C  compatible  and  the  interface  was  modified 
to  facilitate  more  complex  control  scripts.  These  changes  allow  the  program  to  be  more  easily 
operated,  maintained  and  updated.  Higher  level  scripts  were  also  written  that  configure  the 
program  internals  and  interface  via  a  small  number  of  parameters. 

3.  Algorithms 

3.1  Accomplishments 

We  presented  a  paper  at  NIPS*92[4]  on  a  new,  perturbative  method  of  learning  in  feedforward 
(as  well  as  feedback)  networks.  This  described  a  parallel,  stochastic  method  for  learning  in 
neural  networks  without  doing  back-propagation  of  errors.  The  work  focussed  on  a  perturbation 
technique  that  measures,  not  calculates,  the  gradient.  Since  the  technique  uses  the  actual 
network  as  a  measuring  device,  errors  in  modeling  neuron  activation  and  synaptic  weights  do  not 
cause  errors  in  gradient  descent. 

We  focussed  on  the  efficiency  of  one-weight-at-a-time(OWAT)  vs.  parallel  perturbations.  By 
performing  simulation  experiments  on  digital  computers  we  found  that  the  OWAT  gradient 
estimates  were  very  close  to  the  true  gradient  estimates  as  calculated  by  back-propagation.  The 
parallel  perturbation  gradient  estimates  were  always  poorer  than  the  OWAT  estimates,  but  the 
difference  decreased  as  the  number  of  perturbations  increased. 

4.  Preliminary  Research  Definition  -  Real  Time  Adaptive  Electronic  Neural  Classifiers 

4.1  Accomplishments 

We  have  submitted  to  ARPA  (with  Rod  Goodman  and  Pietro  Perona  of  Caltech)  a  proposal  for  a 
multistage  VLSI  learning  and  classification  system  incorporating  some  of  the  above-described 
prototype  hardware  ideas  as  well  as  many  new  ones.  We  are  currently  being  funded  to  study 
these  ideas  in  more  detail  and  develop  a  tightly  defined  proposal. 
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